Yes, that's indeed the case and we're aware of it. It's valid criticism of course. But given the amount of contributions to this project I didn't feel the urge to overcome my laziness in order to do something about it.

About the different approaches:

• Regarding 1: I agree, proofing it's valid Html doesn't seem useful to me. The Html is static after all. The interesting aspect is the JS code for plotly.
• Reg. 2: I've read about selenium, but haven't spent any time trying to understand whether it can help for such a usecase.
• Reg. 3: This is exactly what I had in mind as a first step too, after saving images directly was implemented. But the fact that one would have to update all images if a slight change to default settings is done, was the reason I was too lazy.
• Reg. 4: That actually is a nice idea and I didn't consider it. If the browser is opened successfully and the JS console remains empty, the code should be valid plotly. I might take a closer look at that.

Over the Christmas holidays I actually started experimenting with a different way to proof correctness of the Plotly code. My idea was to look at the Plotly schema:
https://api.plot.ly/v2/plot-schema?sha1=%27%27
and validate the created JSON against that schema at compile time. That experimentation code is here:
https://github.com/Vindaar/MetaPlot
It's basically a macro to write Plotly or Vega-Lite and have the resulting JSON be checked at CT. For Vega-Lite I wrote a JSON Schema validator
https://github.com/Vindaar/JsonSchemaValidator
to validate the JSON properly. Unfortunately Plotly does not provide a JSON Schema. Without it I resorted to some special keys, which need to be replaced, because the actual code is nested etc. seen here:
https://github.com/Vindaar/MetaPlot/blob/master/metaplot.nim#L11-L18
It's really only experimentation, but I feel like if polished it would provide a much nicer way for CI.

• I added approach 5 above, PTAL; any simple idea for the part "that image can be uploaded somewhere" from within CI ? ie, we'd need a server that accepts a POST request to upload that image, to which CI has access to (maybe no permissions needed, it's not security critical)

My idea was to look at the Plotly schema

but, like the html validation approach, this won't catch all bugs; it's useful for unit-testing but not for end-to-end validation; I'm expecting this library to grow more features over time, which also means to javascript logic including things not captured by simply validating the schema

verify the end result seems more robust

• I added approach 5 above, PTAL; any simple idea for the part "that image can be uploaded somewhere" from within CI ? ie, we'd need a server that accepts a POST request to upload that image, to which CI has access to (maybe no permissions needed, it's not security critical)

Sure, this would help somewhat with approach 3. I'd be happy with that for now. I'll comment over at #43 about the subplots.

My idea was to look at the Plotly schema

but, like the html validation approach, this won't catch all bugs; it's useful for unit-testing but not for end-to-end validation; I'm expecting this library to grow more features over time, which also means to javascript logic including things not captured by simply validating the schema

verify the end result seems more robust

Well, if it's a proper schema and the Json is valid under it, there shouldn't be any bugs left. The question is whether the plot actually looks like intended of course. But that's only possible by verifying the end result anyways.